System and method for estimating, synthesizing and matching noise in digital images and image sequences

ABSTRACT

A method for synthesizing noise in a digital image comprises the steps of (a) estimating noise statistics based on a set of spatial autocorrelation coefficients and a set of spectral correlation coefficients that correspond to the color channels of the image, where at least one of the spatial autocorrelation coefficients is weighted by at least one of the spectral correlation coefficients, thereby providing a weighted set of autocorrelation coefficients, and where a set of filter coefficients is obtained from the weighted set of autocorrelation coefficients; and (b) synthesizing a synthetic texture by using the filter coefficients to filter a random noise field in each of the color channels, thereby producing output noise fields in each of the color channels that replicate a synthetic texture, e.g., a desired grain appearance, when combined into the digital image.

FIELD OF THE INVENTION

[0001] The invention relates generally to the field of digital imageprocessing and in particular to digital synthesis and matching of noise.

BACKGROUND OF THE INVENTION

[0002] It is generally known that compositing computer graphics and filmimagery requires that the computer graphics elements be matched in termsof color and texture to the underlying background elements. Typicallythese background elements have been captured on film. It is also knownthat intercutting footage from different sources such as two kinds offilm, or film and digital video, requires that the different elementsmatch to create a seamless look for the entire image sequence. With morefootage from mixed sources and computer graphics, there is a clear needfor a robust grain simulation model. Moreover, there is a need for afully automatic method of estimating grain parameters in image sequencesto reduce the need for user interaction, which can be quite timeconsuming.

[0003] Grain in digitized film images is well characterized by zero-meanGaussian distributed noise for the three color channels. Zero-meanGaussian models are fully characterized by their second-order statistics(the auto-covariance function). Modelling the interactions between colorchannels requires analysis of the spectral correlations. These spectraland spatial correlations can then be used to shape a white noise signalto create synthetic grain. In this way, synthesis of a simulated grainpattern is essentially a texture synthesis problem. But correct shapingof the white noise is required to generate a visually indistinguishable“match” for a given film grain.

[0004] Texture analysis/synthesis methods are well known in theliterature. Bergen and Heeger (“Pyramid-Based TextureAnalysis/Synthesis” SIGGRAPH, 1995, pp. 229-238) describe a method foranalyzing a texture pattern and then replicating it. This method suffersfrom improper color correlation techniques generating significant biasin different color channels. The problem lies with the simple fact thatonce white noise fields have been convolved, the autocorrelationfunction (ACF) is non-zero for each color channel. Adding these togetherusing a color transformation generates the correct spectral correlationsbut biases the spatial correlations. Biased spatial correlations willresult in synthesized textures that look different from the desiredtexture.

[0005] Chellappa and Kashyap considered autoregressive models fortexture synthesis (“Texture synthesis using 2-d noncausal autoregressivemodels,” IEEE Trans. on Acoustics, Speech, and Signal Processing, vol.33, pp. 194-203, February 1985) but the optimization methods required inthat work are inefficient (i.e., long processing times for both analysisand synthesis).

[0006] The newer area of non-parametric synthesis requires templates ofgrain to perform the synthesis and is also known to be morecomputationally intensive than other methods. For example, Efros and T.Leung, (“Texture synthesis by non-parametric sampling,” in ICCV99, pp.1033-1038, 1999) considered how to sample pattern templates to generatenew versions of the pattern. This is time consuming and cannot produce awide enough variety of patterns when considering signal dependent noise.

[0007] With regard to noise estimation, most methods use single frames.Consider, for example, Olsen (“Estimation of Noise in Images: AnEvaluation,” GMIP(55), No. 4, July 1993, pp. 319-323) and Immerkaer,(“Fast Noise Variance-Estimation”, Computer Vision ImageUnderstanding(64), No. 2, September 1996, pp. 300-302). These twomethods estimate noise in generally flat regions of a single image.Motion compensated noise reduction methods are also known in theliterature (see for example Dubois, E., Sabri, S., “Noise Reduction inImage Sequences Using Motion-Compensated Temporal Filtering,” IEEETrans. Communications (32), 1984, pp. 826-831). Although these methodsare used for image sequences, they ignore the concept of iterativelyestimating both motion and noise parameters.

[0008] There are also relevant patents in the prior art. Published U.S.patent application Ser. No. 2002/0034337 (entitled “System forManipulating Noise in Digital images”) discloses a method for shapingwhite noise to create synthetic noise with specific characteristics. Themost obvious problem with this approach is the assumption that the colorchannels can simply be decorrelated and the noise synthesizedindependently. The subband decomposition procedure disclosed thereinfollows closely the treatment by Bergen and Heeger mentioned above.Indeed, it suffers from exactly the same improper color correlationtechniques, thereby generating significant bias in different colorchannels.

[0009] In commonly-assigned U.S. Pat. No. 5,641,596 (which issued Jun.24, 1997 and is entitled “Adjusting Film grain properties in digitalimages” and), D. Cok and R. Gray disclose a method for synthesizing filmgrain using an autoregressive model that yields very good results. Toachieve a better match to older or unknown film stocks, a larger spatialsupport is necessary. The model in that work had a limited spatialsupport and as scanning resolutions increase, spatial support needs togrow accordingly.

[0010] The prior art disclosed above is deficient in its ability toefficiently generate a visually indistinguishable “match” for a givenfilm grain. What is needed is a method of synthesizing noise thatvisually approximates a given film grain and is also close in anumerical sense to the statistics in a given random field. Inparticular, it would be desirable to perform grain matching when giventhe statistics for two different images. Finally, it would be desirableto have a robust method for automatically determining those statisticsfor image sequences.

SUMMARY OF THE INVENTION

[0011] The present invention is directed to overcoming one or more ofthe problems set forth above. Briefly summarized, according to oneaspect of the present invention, the invention resides in a method andsystem for synthesizing noise in a digital image comprised of aplurality of color channels, where the noise is used to represent asynthetic texture that visually approximates a predetermined appearance.More specifically, the method comprises the steps of (a) estimatingnoise statistics based on a set of spatial autocorrelation coefficientsand a set of spectral correlation coefficients that correspond to thecolor channels, where at least one of the spatial autocorrelationcoefficients is weighted by at least one of the spectral correlationcoefficients, thereby providing a weighted set of autocorrelationcoefficients, and where filter coefficients are obtained from theweighted set of autocorrelation coefficients; and (b) synthesizing asynthetic texture by using the filter coefficients to filter a randomnoise field in each of the color channels, thereby producing outputnoise fields in each of the color channels that replicate the synthetictexture when combined into the digital image.

[0012] Accordingly, we disclose a method for synthesizing a texture whengiven second order noise statistics and spectral correlations. This datamay be obtained from scanned film samples, from any digital imagesequence, or may be interactively user controlled. We also disclose amethod for matching noise between two different images using secondorder noise statistics and spectral correlations. Finally, we disclose amethod for automatically computing required noise statistics in imagesequences with or without test target data thus facilitating fullyautomatic robust noise matching.

[0013] The principal advantage of the present invention is that itimproves upon prior art by producing very high quality textures thatresemble grain from film images.

[0014] These and other aspects, objects, features and advantages of thepresent invention will be more clearly understood and appreciated from areview of the following detailed description of the preferredembodiments and appended claims, and by reference to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015]FIG. 1 is a block diagram illustrating the overall workflow forthe method and system according to the invention.

[0016]FIG. 2 is a block diagram illustrating a noise synthesis techniqueby which signal dependent noise is generated.

[0017]FIG. 3 is a diagram illustrating a technique of filter coefficientgeneration by which target spatial autocorrelation parameters aremodified by the spectral correlation coefficients to produce modifiedautocorrelation functions which are subsequently used to solve for thefilter coefficients.

[0018]FIG. 4 is a block diagram illustrating a noise estimationtechnique by which signal dependent noise is estimated directly fromimage data, possibly utilizing an iterative step.

[0019]FIG. 5 is a block diagram illustrating a noise estimationtechnique by which signal dependent noise is estimated directly fromimage data, possibly utilizing an iterative step, with the additionalstep of estimating and using spatial masks to improve the robustness ofthe noise estimation.

[0020]FIG. 6 is a block diagram of an automatic noise-matching algorithmillustrating a technique by which the noise in a source sequence ismatched to the noise in a target sequence.

[0021]FIG. 7 is a block diagram of an automatic noise matching algorithmemploying modification of multiple image sequences, which specificallyillustrates a technique by which noise in one sequence is matched to thenoise in another sequence by modifying both sequences to achieve atarget noise.

DETAILED DESCRIPTION OF THE INVENTION

[0022] Because digital processing systems employing noise estimation,synthesizing and matching are well known, the present description willbe directed in particular to attributes forming part of, or cooperatingmore directly with a system and method in accordance with the presentinvention. System and method attributes not specifically shown ordescribed herein may be selected from those known in the art. In thefollowing description, a preferred embodiment of the present inventionwould ordinarily be implemented as a software program, although thoseskilled in the art will readily recognize that the equivalent of suchsoftware may also be constructed in hardware. Given the system andmethod as described according to the invention in the followingmaterials, software not specifically shown, suggested or describedherein that is useful for implementation of the invention isconventional and within the ordinary skill in such arts. If theinvention is implemented as a computer program, the program may bestored in conventional computer readable storage medium, which maycomprise, for example; magnetic storage media such as a magnetic disk(such as a floppy disk or a hard drive) or magnetic tape; opticalstorage media such as an optical disc, optical tape, or machine readablebar code; solid state electronic storage devices such as random accessmemory (RAM), or read only memory (ROM); or any other physical device ormedium employed to store a computer program.

[0023] The present invention improves upon prior art by producing veryhigh quality textures that resemble grain from film images. Moreover, atechnique for estimating the statistics from an arbitrary sequence isshown and a technique for matching the noise levels from two differentsequences is given.

[0024]FIG. 1 shows the overall system and workflow for the invention asdisclosed herein either for estimating the statistics from an arbitrarysource or sequence (e.g., image sequence 1) or for matching the noiselevels from two (or more) different sequences (e.g., image sequences 1.. . K). In FIG. 1, an image sequence 1 is shown having its noise reducedby a motion compensated filter 10 followed by a noise estimationalgorithm 20. That process may be performed once or may be repeatediteratively (30). At this point the image sequence on a line 32 has hadits noise reduced and a good estimate on a line 34 of both the originaland remaining noise is available. The image sequence may then have noiseadded back to it in an adder stage 40 in order, e.g., to visuallyapproximate a desired appearance, such as the appearance of a given filmgrain.

[0025] The characteristics of the noise that get added back may bedetermined in several ways from an arbitrary source or from one or moreimage sequences. First, the noise may be chosen from predetermined media“looks” or appearances 50 which will make the current sequence look,e.g., in terms of a given film grain, as though it originated on anothermedia. This may be done by analyzing a given media or by having anartist design a particular look and then storing the parameters for thatlook. In this way, infinitely many media “looks” are possible.Alternatively, the noise may be manually controlled by a user in aninteractive control system 60. Finally, the noise may be determined fromthe analysis of the noise estimate returned on line 34 from imagesequence 1 or from an analysis (in an image sequence analysis stage 70)of one or more other digital image sequences (e.g., the image sequenceK). Any of these controls (50, 60, 70) can be used to set the “target”look.

[0026] The noise matching block 80 performs the filter coefficientgeneration and the noise synthesis that determines the characteristicsof the noise that needs to be added to the sequence or sequences in theadders 40, thereby forming an output image sequence 1 (and/or an outputimage sequence K) and/or a composite image sequence output 90. (In FIG.1, the additional “target” image sequence is shown as an image sequenceK, which may be for example a sequence shot against a green screen thatwill be composited with the current image sequence 1. The stage 70 forthe image sequence K includes similar components 10, 20, 30 and 40 forestimating and adding noise to its output image sequence K and/or to thecomposite image sequence output 90.)

[0027] Signal dependent noise that very closely resembles film grain canbe generated using the following model for generating noise:

r(x,y)=α_(r) {tilde over (r)}(x,y) g(x,y)=α_(g) {tilde over(g)}(x,y)+β_(g) r(x,y) b(x,y)=α_(b) {tilde over (b)}(x,y)+β_(b)g(x,y)  Eq. (1)

[0028] This model is illustrated in FIG. 2 for a plurality of randomnoise fields in a plurality of color channels. In that figure, a randomnoise field 100 for the red channel is multiplied in a multiplier 110 bya scale factor α_(r) and filtered in a filter 120 to produce a scaledoutput noise field α_(r){tilde over (r)}(140). Similarly, a random noisefield 102 for the green channel is multiplied in a multiplier 112 by ascale factor α_(g) and filtered in a filter 122 to produce anintermediate noise field α_(g){tilde over (g)}. This is added (adder131) to a portion β_(g) of the red noise field 140, produced by ascaling multiplier 130, to generate a green noise field 142. Finally, arandom noise field 104 for the blue channel is multiplied in amultiplier 114 by a scale factor α_(b) and filtered in a filter 124 toproduce an intermediate noise field α_(b){tilde over (b)}. This is added(adder 133) to a portion β_(b) of the green noise field 142, produced bya scaling multiplier 132, to generate a blue noise field 144. The randomnoise fields 100, 102, 104 are typically Gaussian random noise fieldswhose individual elements are scaled using a noise table describing thevariance at a given code value, i.e., at one or more signal levels. Thisnoise table maps code values to variances or standard deviations, wherethe variances or standard deviations may be a function of the imagedata. Film grain is signal dependent so the underlying image content isneeded to provide appropriate scaling factors (in this work we assumethis scaling is done prior to scaling by the scale factors α_(r), α_(g),and α_(b).) It should be obvious to those skilled in the art thatvarious combinations of the different channels may be added to yield acorrelated color signal.

[0029] Making noise appear as a predetermined texture, such as filmgrain requires that the filtering blocks 120, 122 and 124 shown in FIG.2 produce correct target autocorrelation values and variances. In oneaspect of the invention, a predetermined texture is synthesized whengiven a selected set of spatial autocorrelation coefficients ρ_(R) (N),ρ_(G) (N), ρ_(B) (N) and a selected set of spectral correlationcoefficients ρ_(RG), ρ_(GB), ρ_(RB). This coefficient data may beobtained from conventional sources, such as a calculation from scannedfilm samples or from any digital image sequence, or from interactiveuser control. Based on this coefficient data, it can be shown that thefollowing relations will yield the correct noise statistics in theoutput noise fields 140, 142 and 144: $\begin{matrix}\begin{matrix}{\alpha_{r} = {\alpha_{g} = {\alpha_{b} = 1}}} \\{\sigma_{\overset{\sim}{r}} = \sigma_{r}} \\{\sigma_{\overset{\sim}{g}} = {\sigma_{g}\sqrt{1 - \rho_{rg}^{2}}}} \\{\beta_{g} = {\rho_{rg}\frac{\sigma_{g}}{\sigma_{r}}}} \\{\sigma_{\overset{\sim}{b}} = {\sigma_{b}\sqrt{1 - \rho_{gb}^{2}}}} \\{\beta_{b} = {\rho_{gb}\frac{\sigma_{b}}{\sigma_{g}}}}\end{matrix} & {{Eqs}.\quad (2)} \\{{and},} & \quad \\\begin{matrix}{{\rho_{\overset{\sim}{r}}(N)} = {\rho_{r}(N)}} \\{{\rho_{\overset{\sim}{g}}(N)} = \frac{{\rho_{g}(N)} - {\rho_{rg}^{2}{\rho_{r}(N)}}}{1 - \rho_{rg}^{2}}} \\{{\rho_{\overset{\sim}{b}}(N)} = \frac{{\rho_{b}(N)} - {\rho_{gb}^{2}{\rho_{g}(N)}}}{1 - \rho_{gb}^{2}}}\end{matrix} & {{Eqs}.\quad (3)}\end{matrix}$

[0030] where the notation x(N) indicates a correlation at lag N in onedimension. The equations are also valid when the lag is denoted x(m,n)where m and n denote a lag in the horizontal and vertical direction.This process of reweighting the autocorrelation coefficients isillustrated in FIG. 3 for a multi-channel source of noise data. FIG. 3illustrates how the input autocorrelation functions and spectralcorrelations 200 are used in the reweighting stage 210 according to Eqs.(3) to produce modified autocorrelations 212. Eqs. (3) describes thetarget autocorrelation values that the filtered noise fields should haveso the final noise fields appear as specified when added together. Eqs.(2) describes how to modify the variance (the sigma terms) of the noisefields prior to filtering as well as how to weight the fields whencreating the final synthesized noise (as shown in FIG. 2). With thetarget autocorrelation values (the tilde values) one can build a filterby invoking an optimization algorithm 220 for solving an optimizationproblem. The optimization problem solves for the filter coefficientswhen given the autocorrelation values. The formulation of theoptimization problem depends upon the form of the filter. Filtercoefficients for autoregressive models can be solved using theYule-Walker equations; however, we use a modified linear system tominimize errors in correlation terms at higher lags. As an illustration,for terms up to lag 4 we can solve the following equation in aleast-squares sense: $\begin{matrix}{{\begin{bmatrix}1 & {\rho (1)} & {\rho (2)} & {\rho (3)} \\{\rho (1)} & 1 & {\rho (1)} & {\rho (2)} \\{\rho (2)} & {\rho (1)} & 1 & {\rho (1)} \\{\rho (3)} & {\rho (2)} & {\rho (1)} & 1 \\{\rho (4)} & {\rho (3)} & {\rho (2)} & {\rho (1)} \\0 & {\rho (4)} & {\rho (3)} & {\rho (2)} \\0 & 0 & {\rho (4)} & {\rho (3)} \\0 & 0 & 0 & {\rho (4)}\end{bmatrix}\quad\begin{bmatrix}\theta_{1} \\\theta_{2} \\\theta_{3} \\\theta_{4}\end{bmatrix}} = \begin{bmatrix}{\rho (1)} \\{\rho (2)} \\{\rho (3)} \\{\rho (4)} \\0 \\0 \\0 \\0\end{bmatrix}} & {{Eq}.\quad (4)}\end{matrix}$

[0031] The above steps are repeated for each color channel or band andfor the horizontal and vertical image dimensions to build a separablefilter. For autoregressive models the output is then obtained:

{circumflex over (r)} ₀ =n _(r) ₀ +Σ_(i=1) ^(p)θ_(i) {circumflex over(r)} _(i) {tilde over (r)} ₀ ={circumflex over (r)} ₀+Σ_(i=1)^(p){circumflex over (θ)}_(i) {tilde over (r)} _(i)

ĝ ₀ =n _(g) ₀ +Σ_(i=1) ^(p)ψ_(i) ĝ _(i) {tilde over (g)} ₀ =ĝ ₀+Σ_(i=1)^(p){circumflex over (ψ)}_(i) {tilde over (g)} _(i)

{circumflex over (b)} ₀ =n _(b) ₀ +Σ_(i=1)^(p)X_(i{circumflex over (b)}) _(i) {tilde over (b)} ₀ +{circumflex over(b)} ₀+Σ_(i=1) ^(p){circumflex over (X)}_(i) {tilde over (b)} _(i)  Eqs.(5)

[0032] If a moving average filter is desired, the filter coefficientsmay be determined by solving a nonlinear equation. An example for aone-dimensional non-linear filter with five coefficients is as follows:$\begin{matrix}\begin{matrix}{\min\limits_{\theta_{1},\theta_{2},\theta_{3},\theta_{4}}\left( {{\omega_{1}\left( {{\rho_{\overset{\sim}{r},x}(1)} - \frac{{\theta_{0}\theta_{1}} + {\theta_{1}\theta_{2}} + {\theta_{2}\theta_{3}} + {\theta_{3}\theta_{4}}}{\theta_{0}^{2} + \theta_{1}^{2} + \theta_{2}^{2} + \theta_{3}^{2} + \theta_{4}^{2}}} \right)}^{2} +} \right.} \\{{{\omega_{2}\left( {{\rho_{\overset{\sim}{r},x}(2)} - \frac{{\theta_{0}\theta_{2}} + {\theta_{1}\theta_{3}} + {\theta_{2}\theta_{4}}}{\theta_{0}^{2} + \theta_{1}^{2} + \theta_{2}^{2} + \theta_{3}^{2} + \theta_{4}^{2}}} \right)}^{2} +}} \\{{{\omega_{3}\left( {{\rho_{\overset{\sim}{r},x}(3)} - \frac{{\theta_{0}\theta_{3}} + {\theta_{1}\theta_{4}}}{\theta_{0}^{2} + \theta_{1}^{2} + \theta_{2}^{2} + \theta_{3}^{2} + \theta_{4}^{2}}} \right)}^{2} +}} \\\left. {\omega_{4}\left( {{\rho_{\overset{\sim}{r},x}(4)} - \frac{\theta_{0}\theta_{4}}{\theta_{0}^{2} + \theta_{1}^{2} + \theta_{2}^{2} + \theta_{3}^{2} + \theta_{4}^{2}}} \right)}^{2} \right)\end{matrix} & {{Eq}.\quad (6)}\end{matrix}$

[0033] Similarly, one may build a non-separable autoregressive or movingaverage model using the two dimensional autocorrelation valuesρ_(R)(m,n), ρ_(G)(m,n), ρ_(B)(m,n). These solutions are solved in asimilar way to the methods disclosed above. Whatever the exact solution,the filter coefficients are clearly tied to the modified autocorrelationvalues. These filter coefficients are provided as the output 230 of thefilter coefficient generation process, as shown in FIG. 3, and are thenused as the filter coefficients for the filters 120, 122 and 124 in FIG.2.

[0034] Sometimes the noise statistics of the source and target are knownvia a calibration procedure. More often than not, they are unknown andneed to be estimated from data. Also, it should be clear that arbitrarynoise matching is not always possible unless the source and targetmaterial are modified. Consider simply that one cannot add anuncorrelated random signal to the source material and end up with alower noise position. If the target noise position is lower than thesource position then some form of filtering is required. But noisefiltering usually requires an estimate of signal noise which may beunknown. To accomplish noise estimation and filtering, FIG. 4 shows ameans of iteratively performing signal dependent motion compensatednoise filtering and noise estimation. In FIG. 4, a sequence of sourceimages 300 is filtered by a filtering stage 310 and these filteredimages are compared to the original images by a noise estimationalgorithm 320 to produce an estimate of the noise 330. This process maybe iterated as shown (332) where the new noise estimate is used in themotion estimation and filtering stage 310 to produce a better noiseestimate.

[0035] A preferred motion estimation technique for use with the presentinvention is disclosed in Black, M. J. and Anandan, P., “The RobustEstimation of Multiple Motions: Parametric and Piecewise-SmoothFlow-Fields,” Computer Vision and Image Understanding, January 1996,which is incorporated herein by reference. For the purposes of thepresent discussion a brief overview of the algorithm is given. Black etal. proposed the use of robust functions of the optical flow andsmoothness terms in the general optical flow formulation. Their totalenergy function has the form: $\begin{matrix}{{E_{t}\left( {u,v} \right)} = {{\lambda_{f}{\rho_{f}\left( {{{uI}_{x} + {vI}_{y} + I_{t}},\sigma_{f}} \right)}} + {\sum\limits_{k}{\lambda_{s}{\rho_{s}\left( {{u - u_{k}},\sigma_{s}} \right)}}} + {\lambda_{s}{{\rho_{s}\left( {v - {v_{k}\sigma_{s}}} \right)}.}}}} & {{Eq}.\quad (7)}\end{matrix}$

[0036] The derivatives of the total energy function are: $\begin{matrix}{{\frac{\partial E_{t}}{\partial u} = {{{\lambda_{f}I_{x}{\phi_{f}\left( {{{uI}_{x} + {vI}_{y} + I_{t}},\sigma_{f}} \right)}} + {\sum\limits_{k}{\lambda_{s}{\phi_{s}\left( {{u - u_{k}},\sigma_{s}} \right)}}}} = 0}}{\frac{\partial E_{t}}{\partial v} = {{{\lambda_{f}I_{y}{\phi_{f}\left( {{{uI}_{x} + {vI}_{y} + I_{t}},\sigma_{f}} \right)}} + {\sum\limits_{k}{\lambda_{s}{\phi_{s}\left( {{v - v_{k}},\sigma_{s}} \right)}}}} = 0.}}} & {{Eqs}.\quad (8)}\end{matrix}$

[0037] In these equations ρ is the robust error norm and φ is thederivative of ρ. For example, in Black et al. one robust function usedis the Lorentzian${\rho \left( {x,\sigma} \right)} = {{{\log \left( {1 + {\frac{1}{2}\left( \frac{x}{\sigma} \right)^{2}}} \right)}\quad {with}\quad {\phi \left( {x,\sigma} \right)}} = {\frac{2x}{{2\sigma^{2}} + x^{2}}.}}$

[0038] The selection of the scaling term σ in the above equations isvaried according to the concept of graduated non-convexity, as describedin the paper. However, in the present invention we propose tying theparameter to the signal dependent noise often found in scanned films.Doing this produces more robust motion estimates.

[0039] Given a set of motion estimates, one may apply motion compensatednoise filtering techniques, which are known in the literature to thoseskilled in the art. One means is disclosed in commonly-assigned U.S.Pat. No. 5,600,731 (which issued Feb. 4, 1997 to M. Sezan, M. Ozkan, andS. Fogel, and is entitled “Method for Temporally Adaptive Filtering ofFrames of a Noisy Image Sequence using Motion Estimation”), which isincorporated herein by reference. The filtered image is subtracted fromthe original image to produce a noise map and estimates of signaldependent variance and signal dependent autocorrelation parameters arecomputed using well-known statistical methods. These parameters may thenbe fed back into the motion estimator, as shown by an iterationoperation 332 in FIG. 4, and used to re-compute the motion and filteredimages. Such iteration produces more robust results.

[0040] Occasionally the filtering produces artifacts, and the differencebetween the filtered image and the noisy image contains more than arepresentation of the noise but also signal content. To avoid biasingthe statistics of the noise estimate, we avoid regions with large errorsby using information from the filtering algorithm and/or informationfrom the image itself to generate a spatial mask that may be used tomodify the statistical computation of noise estimates. This procedure isshown in FIG. 5. As before, the entire procedure may be iterated (432)to improve robustness. In FIG. 5, a sequence of source images 400 isfiltered in a motion compensated filtering stage 410 and these filteredimages are compared to the original images in a spatial mask generationstage 420. A spatial mask is generated in the stage 420 to determinewhere good regions are in the image to estimate noise. The filteredimages, original images, and masks are used by a noise estimationalgorithm 430 to produce an estimate of the noise 440. This process maybe iterated (432) as shown where the new noise estimate is used in themotion estimation and filtering stage 410, which produces more reliablemasks and hence better noise estimate.

[0041] The filtered images from a motion compensated noise filter canhave noise added to them to produce matched noise where the matching isdone relative to some other sequence of image data. This matchingprocess is shown in FIG. 6. In FIG. 6, target images 500 and sourceimages 510 have their noise estimated by noise estimation algorithms 520and 530. These noise estimates are used in a filter coefficientgeneration stage 540 to generate filter coefficients. Grain matchingrequires slightly different computations because the intrinsic noisepresent in the source and target image sequences needs to be handled.Eqs. (9) show the model where the subscript “T” denotes “target”, “S”denotes “source”, and “M” denotes “matching”

R _(T) =R _(S)+β_(R) R _(M)

G _(T) =G _(S)+β_(G) G _(M)+_(γG) R _(M)

B _(T) =B _(S)+β_(B) B _(M)+_(γB) G _(M)  Eqs. (9)

[0042] This model differs from our previous model since the sourcematerial has an intrinsic noise level. As before, we solve the weightingparameters: $\begin{matrix}{{\beta_{R} = \sqrt{\frac{\sigma_{R_{T}}^{2} - \sigma_{R_{S}}^{2}}{\sigma_{R_{M}}^{2}}}}{\gamma_{G} = \frac{{\sigma_{R_{T}}\sigma_{G_{T}}\rho_{R_{T}G_{T}}} - {\sigma_{R_{S}}\sigma_{G_{S}}\rho_{R_{S}G_{S}}}}{\beta_{R}\sigma_{R_{M}}^{2}}}{\beta_{G} = {\sqrt{\frac{\sigma_{G_{T}}^{2} - \sigma_{G_{S}}^{2} - {\gamma_{G}^{2}\sigma_{R_{M}}^{2}}}{\sigma_{G_{M}}^{2}}}{\gamma_{B} = \frac{{\sigma_{G_{T}}\sigma_{B_{T}}\rho_{G_{T}B_{T}}} - {\sigma_{G_{S}}\sigma_{B_{S}}\rho_{G_{S}B_{S}}}}{\beta_{G}\sigma_{G_{M}}^{2}}}{\beta_{B} = \sqrt{\frac{\sigma_{B_{T}}^{2} - \sigma_{B_{S}}^{2} - {\gamma_{B}^{2}\sigma_{G_{M}}^{2}}}{\sigma_{B_{M}}^{2}}}}}}} & {{Eqs}.\quad (10)}\end{matrix}$

[0043] Then the desired autocorrelation coefficients of the matchingsignal are: $\begin{matrix}\begin{matrix}{{\rho_{R_{M}}\left( {m,n} \right)} = \frac{{\sigma_{R_{T}}^{2}{\rho_{R_{T}}\left( {m,n} \right)}} - {\sigma_{R_{S}}^{2}{\rho_{R_{S}}\left( {m,n} \right)}}}{\sigma_{R_{T}}^{2} - \sigma_{R_{S}}^{2}}} \\{{\rho_{G_{M}}\left( {m,n} \right)} = \frac{{\sigma_{G_{T}}^{2}{\rho_{G_{T}}\left( {m,n} \right)}} - {\sigma_{G_{S}}^{2}{\rho_{G_{S}}\left( {m,n} \right)}} - {\gamma_{G}^{2}\sigma_{R_{M}}^{2}{\rho_{R_{M}}\left( {m,n} \right)}}}{\sigma_{G_{T}}^{2} - \sigma_{G_{S}}^{2} - {\gamma_{G}^{2}\sigma_{R_{M}}^{2}}}} \\{{\rho_{B_{M}}\left( {m,n} \right)} = \frac{{\sigma_{B_{T}}^{2}{\rho_{B_{T}}\left( {m,n} \right)}} - {\sigma_{B_{S}}^{2}{\rho_{B_{S}}\left( {m,n} \right)}} - {\gamma_{B}^{2}\sigma_{G_{M}}^{2}{\rho_{G_{M}}\left( {m,n} \right)}}}{\sigma_{B_{T}}^{2} - \sigma_{B_{S}}^{2} - {\gamma_{B}^{2}\sigma_{G_{M}}^{2}}}}\end{matrix} & {{Eqs}.\quad 11}\end{matrix}$

[0044] The filter design process follows the exposition given earlierwhere either a separable or nonseparable filter may be used. With thesefilter coefficients, noise is synthesized in a noise synthesis stage 550and added in an adder 552 to the source images resulting in “matched”source images 560 in which the noise position matches that of the targetimages.

[0045] This filtering procedure may also be applied to both sequencesfirst to lower the noise position and then different noise may bedesigned to give both sequences a similar noise position. Each sourcesequence may have a different source noise level. By choosing a similartarget noise level, the matched noise added to each sequence will bedifferent but the final noise positions will be the same. This method isshown in FIG. 7. In FIG. 7, image sequence 600 (sequence 1) is filteredby a motion compensated noise filter 602 and then the noise is estimatedby a noise estimation algorithm 604, with this process repeatediteratively (605) as desired. Filter coefficients are created in afilter generation stage 606 and noise is synthesized in a noisesynthesis stage 608 and added to the filtered images in a summationstage 609 to produce a matched imaged sequence 610 (sequence 1).

[0046] Similarly in FIG. 7, image sequence 620 (sequence 2) is filteredin a motion compensated noise filter 622 and then the noise is estimatedby a noise estimation algorithm 624, with this process repeatediteratively (625) as desired. Filter coefficients are created in afilter coefficient generation stage 626 and noise is synthesized in anoise synthesis stage 628 and added to the filtered images to produce amatched imaged sequence 630 (sequence 2) in a summation stage 629. Thefilter coefficients are tied together because the noise in each sequenceis coupled to create the same “noise look” in both sequences. This isthe same as the noise matching shown in FIG. 1. Although the sourcenoise level is different in image sequences 1 and 2, by manipulating thefilter coefficients in generation stages 606 and 626 to achieve asimilar target noise level, the matched noise added to each sequence insummation stages 609 and 629 will be different but the final noisepositions of the sequences 1 and 2 will be the same. Thereupon, aconventional compositing operation 80 (FIG. 1) may be performed on thematched sequences. The filter coefficient generation may be doneautomatically or may include some degree of manual intervention to guidethe final look for each image sequence.

[0047] The invention has been described in detail with particularreference to certain preferred embodiments thereof, but it will beunderstood that variations and modifications can be effected within thespirit and scope of the invention.

Parts List

[0048]10 motion compensated filter

[0049]20 noise estimation algorithm

[0050]30 iteration

[0051]32 line

[0052]34 line

[0053]40 adder stage

[0054]50 predetermined media “looks”

[0055]60 user control system

[0056]70 noise matching block

[0057]80 composite output

[0058]100 random noise field for red channel

[0059]102 random noise field for green channel

[0060]104 random noise field for blue channel

[0061]110 red scaling factor multiplier

[0062]112 green scaling factor multiplier

[0063]114 blue scaling factor multiplier

[0064]120 filter for red channel

[0065]122 filter for green channel

[0066]124 filter for blue channel

[0067]130 red/green inter-channel scaling multiplier

[0068]131 adder

[0069]132 green/blue inter-channel scaling multiplier

[0070]133 adder

[0071]140 output red noise

[0072]142 output green noise

[0073]144 output blue noise

[0074]200 input spectral correlations and autocorrelation functions

[0075]210 reweighting stage

[0076]212 modified autocorrelation functions

[0077]220 optimization algorithm

[0078]230 filter coefficients

[0079]200 input spectral correlations and autocorrelation functions

[0080]300 source images

[0081]310 motion compensated filtering

[0082]320 noise estimation algorithm

[0083]330 noise data

[0084]332 iteration

[0085]400 source images

[0086]410 motion compensated filtering

[0087]420 spatial mask generation

[0088]430 noise estimation algorithm

[0089]432 iteration

[0090]440 noise data

[0091]500 target images

[0092]510 source images

[0093]520 noise estimation algorithm

[0094]530 noise estimation algorithm

[0095]540 filter coefficient generation

[0096]550 noise synthesis

[0097]552 adder

[0098]560 matched source images

[0099]600 image sequence 1

[0100]602 motion compensated noise filtering

[0101]604 noise estimation algorithm

[0102]605 iteration

[0103]606 filter coefficient generation

[0104]608 noise synthesis

[0105]609 summation stage

[0106]610 matched image sequence 1

[0107]620 image sequence 2

[0108]622 motion compensated noise filtering

[0109]624 noise estimation algorithm

[0110]625 iteration

[0111]626 filter coefficient generation

[0112]628 noise synthesis

[0113]629 summation stage

[0114]630 matched image sequence 2

What is claimed is:
 1. A method for synthesizing noise in a digitalimage comprised of a plurality of color channels, where the noise isused to represent a synthetic texture that visually approximates apredetermined appearance, said method comprising the steps of: (a)estimating noise statistics based on a set of spatial autocorrelationcoefficients and a set of spectral correlation coefficients thatcorrespond to the color channels, where at least one of the spatialautocorrelation coefficients is weighted by at least one of the spectralcorrelation coefficients, thereby providing a weighted set ofautocorrelation coefficients, and where filter coefficients are obtainedfrom the weighted set of autocorrelation coefficients; and (b)synthesizing a synthetic texture by using the filter coefficients tofilter a random noise field in each of the color channels, therebyproducing output noise fields in each of the color channels thatreplicate the synthetic texture when combined into the digital image. 2.The method as claimed in claim 1 wherein the step (b) of synthesizing asynthetic texture comprises producing a filtered noise field in each ofthe channels and weighting at least one of the filtered noise fieldswith a portion of another filtered noise field to produce at least oneof the output noise fields.
 3. The method as claimed in claim 1 whereinthe noise statistics estimated in step (a) are based on a set of spatialautocorrelation coefficients and a set of spectral correlationcoefficients obtained from an arbitrary source.
 4. The method as claimedin claim 3 wherein the arbitrary source is obtained from a plurality ofpredetermined media “looks” or appearances.
 5. The method as claimed inclaim 3 wherein the arbitrary source is originated from userinteraction.
 6. The method as claimed in claim 1 wherein the digitalimage comprises a sequence of digital images and the noise statisticsestimated in step (a) are based on a set of spatial autocorrelationcoefficients and a set of spectral correlation coefficients obtainedfrom an analysis of the sequence of digital images.
 7. The method asclaimed in claim 6 wherein the analysis of the sequence of digitalimages comprises the steps of motion compensated filtering of thesequence of digital images, thereby providing filtered images, andoperation on the filtered images with a noise estimation algorithm toprovide the noise statistics.
 8. The method as claimed in claim 7further comprising the step of spatially masking the filtered images toidentify preferred images areas for noise estimation.
 9. The method asclaimed in claim 1 wherein the digital image comprises a sequence ofsource images and another sequence of target images, and wherein thestatistics estimated in step (a) are based on a set of spatialautocorrelation coefficients and a set of spectral correlationcoefficients obtained from analyzing and matching the source images andthe target images.
 10. The method as claimed in claim 1 wherein thedigital image comprises a first sequence of images and a second sequenceof images, each with a different source noise level, and wherein thestatistics estimated in step (a) are obtained for each sequence ofimages by motion filtering of each sequence, thereby providing filteredimages in each sequence, and operation on the filtered images in eachsequence with a noise estimation algorithm to provide the noisestatistics, wherein the noise statistics in each sequence are adjustedto a similar target noise level, whereby the output noise field added toeach sequence will be different but the final noise positions of thesequences will be the same.
 11. The method as claimed in claim 1 whereinthe synthetic texture visually approximates a desired grain appearance.12. A method for generating filter coefficients for use in noisesynthesis from a multi-channel source of noise data, said methodcomprising the steps of: (a) providing a set of target autocorrelationfunction (ACF) values for one or more signal levels for each channel ofthe multi-channel noise data and a set of target spectral correlationsfor the multi-channel noise data; (b) modifying the target ACF valuesusing the target spectral correlation coefficients to produce a set ofmodified ACF values; and (c) generating filter coefficient values forthe filter coefficients from an optimization algorithm by using themodified ACF values to solve an optimization problem defined by theoptimization algorithm.
 13. The method as claimed in claim 12 wherein anoise table is used in step (a) to specify the variance of the one ormore signal levels for each channel of the multi-channel noise data. 14.The method as claimed in claim 12 wherein the filter coefficientsgenerated in step (c) are used to modify the noise characteristics of amulti-channel digital image signal, said method further comprising thesteps of: (d) providing a random noise field in each of the channel; (e)using the filter coefficients to filter each random noise field in eachchannel, thereby producing a plurality of filtered noise fields; (f)weighting one or more of the filtered noise fields, thereby producingfinal noise fields in each of the signal channels; and (g) adding thefinal noise fields to the channels of the digital image signal.
 15. Themethod as claimed in claim 14 wherein the noise fields provided in step(d) have a variance that is a function of the digital image signal. 16.A method for generating noise that visually approximates a desiredappearance when added to a multi-channel digital image, where the noiseis synthesized from a multi-channel source of noise data, said methodcomprising the steps of: (a) providing a set of target autocorrelationfunction (ACF) values for one or more signal levels for each channel ofthe multi-channel noise data and a set of target spectral correlationsfor the multi-channel noise data; (b) modifying the target ACF valuesusing the desired spectral correlation coefficients to produce a set ofmodified ACF values; (c) generating filter coefficient values for thefilter coefficients from an optimization algorithm by using the modifiedACF values to solve an optimization problem defined by the optimizationalgorithm (d) using the filter coefficients to filter a random noisefield provided in each signal channel, thereby producing a plurality offiltered noise fields; (e) weighting one or more of the filtered noisefields, thereby producing final noise fields in each of the signalchannels; and (f) adding the final noise fields to the channels of thedigital image signal.
 17. The method as claimed in claim 16 wherein anoise table is used in step (a) to specify the variance of the one ormore signal levels for each channel of the multi-channel noise data. 18.The method as claimed in claim 16 wherein the random noise fieldsprovided in step (d) have a variance that is a function of the digitalimage signal.
 19. The method as claimed in claim 16 wherein theweighting of one or more of the filtered noise fields in step (e)comprises weighting a filtered noise field in one channel with a portionof another filtered noise field from another channel.
 20. The method asclaimed in claim 16 wherein the desired appearance being approximatedcomprises the appearance of a predetermined film grain.
 21. A method forestimating the noise statistics in an image sequence comprising aplurality of source image frames, said method comprising the steps of:(a) finding a correspondence between the source image frames in theimage sequence based on motion estimates between the frames; (b) using amotion-compensated filter to generate an estimate for one or more noisefree image frames in the image sequence; (c) using the difference of thenoise free image frames and the source image frames to create noiseimage data for a noise image frame; (d) iterating the noise image databack through the motion-compensated filter to improve the noise imagedata and the noise image frame; and (e) using the noise image frame fromstep (d) to compute noise statistics.
 22. The method as claimed in claim21 further comprising the steps of: (f) generating a spatial mask usingat least one of the source image, the filtered image and the noiseimage, said mask being used to identify candidate regions within thesource image frames with valid noise estimates; and (g) using theidentified candidate regions from step (f) in combination with step (e)to compute noise statistics.
 23. A method for matching noise between asource and target digital image, where the noise is synthesized from amulti-channel source of noise data, said method comprising the steps of:(a) providing a set of source and target autocorrelation function (ACF)values for one or more signal levels for each channel of themulti-channel noise data and a set of source and target spectralcorrelations for the multi-channel noise data; (b) modifying the sourceand target ACF values using the source and target correlationcoefficients to produce a set of modified ACF values; (c) generatingfilter coefficient values for the filter coefficients from anoptimization algorithm by using the modified ACF values to solve anoptimization problem defined by the optimization algorithm (d) using thefilter coefficients to filter a random noise field provided in eachsignal channel, thereby producing a plurality of filtered noise fields;(e) weighting one or more of the filtered noise fields, therebyproducing final noise fields in each of the signal channels; and (f)adding the final noise fields to the channels of the digital imagesignal to produce an image having noise statistics of the target image.24. A system for synthesizing noise in a digital image comprised of aplurality of color channels, where said noise is used to represent asynthetic texture that visually approximates a predetermined appearance,said system comprising: (a) a noise estimation stage for estimatingnoise statistics based on a set of spatial autocorrelation coefficientsand a set of spectral correlation coefficients that correspond to thecolor channels, where at least one of the spatial autocorrelationcoefficients is weighted by at least one of the spectral correlationcoefficients, thereby providing a weighted set of autocorrelationcoefficients, and where a set of filter coefficients are obtained fromthe weighted set of autocorrelation coefficients; and (b) a noisesynthesis stage for synthesizing a synthetic texture by using the filtercoefficients to filter a random noise field in each of the colorchannels, thereby producing output noise fields in each of the colorchannels that replicate the synthetic texture when combined into thedigital image.
 25. The system as claimed in claim 24 wherein the noisesynthesis stage produces a filtered noise field in each of the channelsand weights at least one of the filtered noise fields with a portion ofanother filtered noise field to produce at least one of the output noisefields.
 26. The system as claimed in claim 24 wherein the noisestatistics estimated by the noise estimation stage are based on a set ofspatial autocorrelation coefficients and a set of spectral correlationcoefficients obtained from an arbitrary source.
 27. The system asclaimed in claim 26 wherein the arbitrary source is obtained from aplurality of predetermined media “looks” or appearances.
 28. The systemas claimed in claim 26 wherein the arbitrary source is originated fromuser interaction.
 29. The system as claimed in claim 24 wherein thedigital image comprises a sequence of digital images and the noisestatistics estimated by the noise estimation stage are based on a set ofspatial autocorrelation coefficients and a set of spectral correlationcoefficients obtained from an analysis of the sequence of digitalimages.
 30. The system as claimed in claim 29 wherein the noiseestimation stage includes a filter for motion compensated filtering ofthe sequence of digital images, thereby providing filtered images, and anoise estimation algorithm for operating on the filtered images toprovide the noise statistics.
 31. The system as claimed in claim 24wherein the noise estimation stage further includes an algorithm forspatially masking the filtered images to identify preferred images areasfor noise estimation.
 32. The system as claimed in claim 24 wherein thedigital image comprises a sequence of source images and another sequenceof target images, and wherein the statistics estimated by the noiseestimation stage are based on a set of spatial autocorrelationcoefficients and a set of spectral correlation coefficients obtainedfrom analyzing and matching the source images and the target images. 33.The system as claimed in claim 24 wherein the digital image comprises afirst sequence of images and a second sequence of images, and whereinthe statistics estimated by the noise estimation stage are obtained foreach sequence of images by motion filtering of each sequence, therebyproviding filtered images in each sequence, and operation on thefiltered images in each sequence with a noise estimation algorithm toprovide the noise statistics, wherein the noise statistics in eachsequence are adjusted to a similar target noise level.